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Abstract — We use the replica method of statistical mechanics 
to examine a typical performance of correctly reconstructing N- 
dimensional sparse vector x — (xt) from its linear transforma- 
tion y — Fx of P dimensions on the basis of minimization 
of the Lp-norm ||a;||p = lim e _> + o YliLi \ x A v+t - We characterize 
the reconstruction performance by the critical relation of the 
successful reconstruction between the ratio a = P/N and the 
density p of non-zero elements in x in the limit P, N — > oo 
while keeping a ~ 0(1) and allowing asymptotically negligible 
reconstruction errors. We show that the critical relation a c (p) 
holds universally as long as F T F can be characterized asymp- 
totically by a rotationally invariant random matrix ensemble and 
FF T is typically of full rank. This supports the universality of 
the critical relation observed by Donoho and Tanner (Phil. Trans. 
R. Soc. A, vol. 367, pp. 4273-4293, 2009; arXiv: 0807.3590) for 
various ensembles of compression matrices. 

I. Introduction 

Let us suppose a situation that an TV-dimensional vector 
x° G R N is linearly transformed into a P-dimensional vector 
y G K p , where y = Fx, by using a P x N matrix 
F G R PxN . When a = P/N < 1, it is generally impossible 
to reconstruct the original vector x° from y and F correctly. 
However, if x° is sparse, which means that the fraction p 
of non-zero elements in a; is less than unity, a correct 
reconstruction may be possible by utilizing prior knowledge of 
the sparsity. The framework for making efficient use of such a 
possibility is often termed compressed (compressive) sensing 
0. The research history of this scheme is rather long and dates 
back to the 1970s 0, 0. However, since the publication of 
a series of influential papers in the middle of the 2000s, the 
horizon of this field has rapidly expanded 0,0,0,0,0. 

For exploiting the sparsity in reconstructing x° from y given 
F , reconstruction schemes based on minimization of the so- 
called Lp-norm of x = (xi) G 
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the number of nonzero elements in a;, p = 

under a linear constraint Fx = y have been actively ex- 
plored. We will hereafter refer to these schemes as L p - 
reconstruction. In particular, much attention has been paid 



to L\ -reconstruction since (HJ under the linear constraint can 
be minimized by using convex optimization methods with a 
computational cost that is polynomial in N for p = 1 fl9l, 

una, on, ma- 

The following result on L\ -reconstruction was reported 
recenly 0, 0, lfl3l . Let us suppose that the number of 
non-zero elements in x° is upper-bounded by S. Let us also 
assume that F is composed of independently and identically 
distributed (i.i.d.) Gaussian random variables of zero mean 
and a fixed variance. In addition, we shall consider that the 
reconstruction is successful if and only if the reconstructed 
vector x exactly accords to x°. Accordingly, if the two 
inequalities 
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hold simultaneously, the probability of the L\ -reconstruction 
failing vanishes as N tends to infinity. The previous studies 
lfT4l . |fT31 obtained the critical relations between a and p for 
typical and worst cases when a; is randomly generated under 
the same assumption of F and for the same criterion on the 
successful reconstruction as P, N — > oo while maintaining 
a - 0(1). 

The above two results have one thing in common in the 
sense that the critical relations are evaluated under a success 
criterion that allows no reconstruction errors. However, in 
performance evaluations of large systems, it may be more 
plausible to permit asymptotically negligible errors in char- 
acterizing the success of reconstruction. In addition, it may 
be natural to ask how good a reconstruction can be obtained 
by using the schemes of p ^ 1 and/or other choices of 
compression matrix F. 

The purpose of this paper is to answer these questions. 
More specifically, we utilize the replica method of statistical 



mechanics to explore the typical performance of the L p - 
reconstruction while allowing asymptotically negligible re- 
construction errors in the large system limit for a class of 
compression matrices lfl6l . 

This paper is organized as follows. The next section intro- 
duces the model that we will examine. Section III outlines the 
analysis. In section IV, the results of the analysis are shown in 
conjunction with an experimental validation. The final section 
is devoted to a summary. 

II. Model definition 

We will consider the following simple scenario of com- 
pressed sensing. Each component of the original vector x°, 



is i.i.d. following a distribution 

P(x) = (1 - p)S(x) + pr{x) 



(4) 



where 8(x) denotes Dirac's 5 function and r(x) is an arbitrary 
distribution with finite first- and second-order moments and 
without a finite mass at the origin. For simplicity, we assume 
that the second moment about the origin is unity without 
loss of generality. Motivated by the observed universality 
of the critical relation between a and p for the successful 
reconstruction, which was reported by earlier studies ifTTl . |[T8l 
for various compression matrices, we characterize F as 

F = UDV T (5) 

utilizing a form of singular value decomposition |fl9l , where 
U and V are samples of uniform distributions over P x P 
and TV x TV orthogonal matrices, respectively, which are 
independent of each other and of D. T denotes the matrix 
transpose operation. D is a P x TV diagonal matrix such 
that the eigenvalues of the P x P matrix DD T (or FF T ) 
asymptotically follow a certain distribution /(A), the support 
of which is defined over a bounded interval in A > as 
P = aN tends to infinity. The situation in which F is 
composed of i.i.d. random variables of zero mean and a fixed 
variance corresponds to the case that /(A) is provided by a 
distribution of the Marchenko-Pastur type 1201 . Let us suppose 
that data m° (which may represent image, sound, etc.) is 
expressed as m° = V T x° by utilizing an orthogonal basis 
V T and a sparse coefficient vector x°. The assumption of 
(O corresponds to a situation in which x° is inferred from 
projections of m° to randomly chosen P orthogonal directions 
specified by U in conjunction with certain signal amplification 
expressed by D. 

In the following, we will examine the critical relation 
between a and p for correctly reconstructing x° from its 
compressed expression y = Fx by the L p -reconstruction, 
in particular, for cases of p = 0, 1 and 2. However, such a 
critical relation disappears when y is smeared by noise of 
non-negligible strength l2T1 . 

III. Outline of Replica Analysis 
A posterior distribution 

r^"W'S(Fx - y) 
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given a compressed expression y and compression matrix 
F is the basis of our assessment. The normalization factor 
Z(/3;y) = J e~^ m '' p 8(Fx — y)dx acts as the partition 
function in statistical mechanics. In the limit /3 — s- oo, the 
posterior distribution (O converges to a uniform distribution 
over solutions of the L p -reconstruction which minimize ||a;|| p 
under the constraint Fx = y. Therefore, one can explore 
the performance of the L p -reconstruction by examining the 
properties of the posterior distribution © in the limit f3 — > oo. 

A distinctive feature of the posterior distribution © is that it 
depends on predetermined (quenched) randomness y(= Fx ) 
and F, which naturally leads us to employ the replica method 
El . More precisely, we evaluate the generating function of 
the partition function $(ra,/3) = TV -1 In [Z n (f3; y)} (n € R) 
by analytically continuing the functional expressions obtained 
for n — 1, 2, . . . e N to n G M, where [• • ■] denotes averaging 
with respect to the predetermined random variables y and F 
(or Xq, U and V). Once the expression of (3) is obtained, 
typical properties of the L p -reconstruction can be examined 
by evaluating the minimized L p -norm (per element), C p , with 
the identity C p = — lim^oo lim„_j.o f3~ 1 (d/dn)$(n, f3). 
Similar schemes have been employed for analyzing various 
problems concerning information and communication l23l . 
Il24l including that of noisy sensing I2TI . 

For n G N, expanding the n-th power of the integral of the 
partition function yields 
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Employment of the techniques developed in J25J, 
ll26l . in conjunction with a heuristic identity 

S(x) = lim T ^ + o(27TT) _1 ' /2 cxp(-x 2 /(2r)), leads to 
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Here, [•••][/ v denotes an average with respect to U and V, 
and G(x) is defined on the basis of the Shannon transform of 
the eigenvalue distribution /(A), as 



G{-x) = cxtr|-||/(A)ln(A + A)rfA + ^ 
1 . 1 
2' 



In: 
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where extrx(- • •) means extremization with respect to X. For 
x 3> 1, this function asymptotically behaves as G(— x) ~ 
— (a/2)lna; + const, irrespectively of /(A) as long as the 
support of /(A) is defined over a bounded interval in A > 0. 
T(n) is an n x n matrix whose elements are T a b = N~ 1 (x a — 
x°) ■ (x b — x°) for a, b = 1,2, ...,n. Under the replica 
symmetric (RS) ansatz TV _1 |a; a | 2 = Q, N~ 1 x a ■ x b = q 
and N~ 1 x° ■ x a = m for b) = 1,2, ... ,n, T{n) has 
eigenvalues Q — q — n(Q — 2m + Q°) and Q — q of a 
single degeneracy and n — 1 denegeracies, respectively, where 



Q° = JV — 1 |a3° | 2 . These make it possible to define the right- 
hand side of dH) for n G K. Combining this and the expression 
for the volume of variables {x a } that satisfy the RS ansatz, 
which can be assessed by using the saddle-point method and 
taking an average with respect to x°, finally yields 
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where 6 = {Q, \, m, Q, x, m}, x 
ex.p(—z 2 /2)dz/\/2ir, and 
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We have used the fact Q° — >• p J r(x°)(x ) 2 dx° = p, which 
follows from our assumption on the distribution P(x). 

At this point, we should note the following issues. Firstly, 
( fTOl depends on no properties of D except for the compression 
rate a. This means that the asymptotic performance of the L p - 
reconstruction is unchanged as long as F T F can be asymptot- 
ically characterized by a rotationally invariant matrix ensemble 
and FF T is typically of full rank. This is in accordance with 
the universality on the reconstruction thresholds for various 
matrix ensembles observed in ifPTl . ED. Such universality 
is, however, limited to the noiseless cases; the reconstruction 
performance does depend on D when non-negligible noises 
exist 121]. Secondly, Q and m determined by (TToT > represent 
values of TV" 1 [|S| 2 ] and iV" 1 [x° ■ sc], respectively, as long 
as the solution of the L p -reconstruction x is uniquely deter- 
mined in typical cases. This means that the typical value of the 
mean square error (MSE) per element between the original and 
reconstructed vectors, £ = N^ 1 [\x — cc | 2 ] , can be assessed 
as 



£ = Q- 2m + p 



(12) 



by using the extremum solution of (TTOb . When x typically 
accords with a; , Q = m = p holds, yielding £ = 0. 
Therefore, the critical condition between a and p for the L p - 
reconstruction, which can be expressed as the critical rate of 
the reconstruction limit a c (p), can be obtained by examining 
the stability of the successful solution Q = m = p of 
(fTOl . Thirdly, asymptotically negligible MSEs are allowed 
for successful reconstruction in the current analysis, whereas 
no errors are permitted in typical case analyses presented 
in earlier studies 0141 . Ifl5l . This means that our assessment 
is of utility for examining how sensitive the performance is 
to the criterion of the success of reconstruction. Finally, we 
must keep in mind that the RS ansatz that is adopted in the 
current analysis is not necessarily guaranteed to be correct. 
For instance, the local stability of the RS saddle point is lost 
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Fig. 1. x*(h; Q) for p = 
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(h - h/\h\)/Q for 
(c): x* 2 {h;Q) = h/(Q + 2). 



against perturbations that break the replica symmetry if 
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holds |27l. Here, x* p {h;Q) = -(d/dh)<f> p (h; Q), the profile 
of which is shown in Fig. Q] for p = 0, 1 and 2. When 
the condition < fTf3T > holds for the extremum solution of ( fTUl ), 
the RS treatment is not valid and one has to explore more 
general solutions taking into account the effect of replica 
symmetry breaking (RSB) in order to accurately assess the 



Lp-reconstruction. 



IV. Results 



A. Theoretical predictions 

We numerically solved the saddle-point problem ( flOb for 
various pairs of a and p for p = 0, 1 and 2. When p and p are 
fixed, we always found only the successful solution Q = m = 
p for sufficiently large a. As we decreased a, the successful 
solution lost its stability within the RS ansatz and exhibited a 
transition to a failure solution. 

For the successful solution, conjugate variables Q and fh 
are always infinitely large. On the other hand, x an d X do 
not necessarily diverge. We assessed the lower limit of the 
compression rate for given p, a c (p), for the L p -reconstruction 
by examining the local stability of the successful solution. 

1) p = 0: The successful solution is stable if and only if 
a > p. This indicates that a c (p) = p for p = 0. Even if the 
positions of the non-zero elements in x° are known, a > p 
is necessary for Fx = y to have a unique solution. This 
implies that the Lq -reconstruction achieves the best possible 
performance of compressed sensing. However, the condition 
( TT3b always holds for the successful solution because of 
the discontinuous profile of x*(h;Q). This means that the 
assessment under the RS ansatz is not appropriate and further 
explorations taking RSB into account will be necessary before 
an accurate performance evaluation of the Lo-reconstruction 
can be made within our framework. However, this is beyond 
the scope of the current study. 



2) p — 1: x of the successful solution is determined by 
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where Q(.t) = f°° is the conventional Q-function. Utiliz- 
ing the solution of this equation, the stability condition of the 
successful solution is expressed as 



a > 2(1- p)Q{x 



-1/2 > 



(15) 



indicating that the reconstruction limit of the L\ -reconstruction 
is a c (p) = 2(1 — p)Q(x~ 1 ) +P- a c{p) a ls° accords with the 
critical condition of (fTST l. which implies that the RS solution 
is stable as long as ( TT3T > holds. We thus conclude that there is 
no need of performing the RSB analysis for the performance 
evaluation of the L\ -reconstruction. 

We should emphasize that the reconstruction limit (fT5T > 
accords exactly with the limit obtained in lfl4l . Ifl5ll under 
the criterion of a successful reconstruction that permits no 
reconstruction errors of any order by utilizing techniques of 
combinatorial geometry. This indicates that the reconstruc- 
tion limit is not sensitive to the criterion of the successful 
reconstruction in the sense that it does not change even if 
we allow asymptotically negligible reconstruction errors as 
P = aN -t oo. 

Our results also imply that the replica method may be of 
utility not only to the current problem of compressed sensing 
but also to the whole field of combinatorial geometry. 

3) p = 2: The successful solution is stable if and only if 
a > 1, which indicates that a c (p) = 1. For a > a c {p) = 
1, ( TOI l does not hold and, therefore, there is no need to do 
the RSB analysis. However, for a > 1, one can perfectly 
reconstruct x° by using the Gauss elmination method to solve 
the linear equation Fx = y. This leads us to conclude that the 
^-reconstruction has no capability of compressed sensing. 

B. Consideration 

Figure |2] (a) shows a c (p) for p = 0, 1 and 2. For compari- 
son, the inset shows plots of (01 and ® in the limit of N, P — > 
oo while maintaining a = P/N ~ 0(1), p = S/N ~ 0(1). In 
the case of large systems, the L% -reconstruction is performable 
with a computational cost of 0(N 3 ) by utilizing interior point 
methods. On the other hand, although the Lo- recons truction 
may potentially give a better reconstruction capability than 
the L\ -reconstruction, it is NP-hard. Figure |2] (a) and these 
indicate that the Li-reconstruction is a practical method that 
possesses both computational feasibility and relatively high 
reconstruction capability. 

There is a huge discrepancy between the typical reconstruc- 
tion limit obtained here and the one for the worst case of 
@, (D, lfl3l . which is shown in the inset. This implies that 
there may be much room for improvement of the worst-case 
assessment, although we must keep in mind that the criterion 
of reconstruction success in the current analysis, which permits 
asymptotically negligible reconstruction errors as N — s- oo, 
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Fig. 2. (a): Typical reconstruction limit for the L p -reconstruction of 
p = 0, 1 and 2. The inset represents the worst case upper-bound for 
the L\ -reconstruction offered by (2) and (3)- (b): Experimental assessment 
of a c (p = 0.5) for the L\ -reconstruction for (A): matrices of i.i.d. 
Gaussian random entries (circle) and (B): those of random orthogonal row 
vectors (triangle). A quadratic fitting with respect to 1/JV yields a c (0.5) = 
limjv-voo Qc(0.5, N) ~ 0.83165 and 0.83179 for (A) and (B), respec- 
tively. 



is different from that of the worst-case analysis in which no 
errors are allowed. 

C. Experimental validation 

To justify the result obtained above, we carried out nu- 
merical experiments on the L\ -reconstruction and the original 
vectors of p = 0.5 for two matrix ensembles; (A): matrices 
of i.i.d. Gaussian random entries and (B): those of random 
orthogonal row vectors. The results are summarized in Fig. |2] 
(b). For both ensembles and each of N = 10, 12, . . . , 30, 
we numerically evaluated the critical compression rate for 
successful reconstruction for p = 0.5, a c (p = 0.5, N), on the 
basis of 10 6 experiments. We employed a package of convex 
optimization schemes CVX ll28l . ll29l for searching x. The 
critical values of N — s- oo were estimated by extrapolation 
of a quadratic fitting with respect to iV _1 , which yielded 
a c (0.5) = limjv--,^ a c (0.5, N) ~ 0.83165 and 0.83179 for 
(A) and (B), respectively. These values are in accordance with 
the theoretical prediction a c (0.5) = 0.83129... up to the 
third digit (Fig. 12(b)), which supports the universality of the 
critical relation. In @, the critical density p c for a — 0.5 
is experimentally evaluated for relatively large systems of 



N = 512 and 1024 in the case of the i.i.d. Gaussian random 
entries. Although the exact value is not provided, judging 
by eye, the plots of the results are close to our theoretical 
prediction p c (a = 0.5) = 0.192 84 . . .. This indicates that our 
approach at least has the ability to predict these experimental 
results with high accuracy. However, a general mathematical 
justification of the replica method is still to be done 1301 . 

V. Summary 

In summary, we assessed a typical reconstruction limit 
of compressed sensing based on the L p -norm reconstruction 
under linear constraints for p = 0, 1 and 2 in the limit of large 
systems utilizing the replica method of statistical mechanics. 
For rotationally-invariant ensembles of the compression matrix 
F, our analysis indicates that the followings hold univer- 
sally; the replica symmetric solution of the Ln-reconstruction 
achieves theoretically optimal reconstruction performance, but 
unfortunately, is unstable against perturbations that break the 
replica symmetry, Consequently, further analysis based on 
the replica symmetry breaking ansatz will be necessary for 
accurate assessment of its performance. The -reconstruction 
has no capability of compressed sensing. On the other hand, 
the L\ -reconstruction has good reconstruction ability which is 
characterized by a universal critical relation between a and p. 
Moreover, it is widely known that the L\ -reconstruction can 
be done at a computationally feasible cost. These properties 
point to the great utility of the L\ -reconstruction in practice. 

Our analysis supports that the universality of the critical 
relation observed in lfT7l . lfl8l holds for a certain class of the 
matrix ensembles irrelevantly to the details of the distribution 
of non-zero elements in the original vector x°. However, the 
relations can depend on the details of F when the ensemble 
is not rotationally invariant: this situation practically occurs 
when the original information to be compressed is expanded 
by non-orthogonal bases and statistically independent sparse 
coefficients. An analysis on such cases will be reported else- 
where ll3D . 
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